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Abstract 

In  flightline  maintenance,  collaboration 
is  an  important  factor  in  the  maintainer’s 
quest  to  complete  a  difficult  or  unusual 
repair.  Technology  provides  a  multitude 
of  potential  collaborative  tools  and 
techniques,  and  this  must  be  balanced 
against  the  requirement  to  leverage 
and/or  support  maintained  existing 
interaction  skills,  rather  than  requiring 
them  to  adapt  to  the  technology.  This 
paper  discusses  the  definition  of 
collaboration,  a  high-level  classification 
of  collaboration  types,  followed  by  a 
discussion  of  the  strategy  used  by  the 
Aircraft  Maintenance  Intuitive 
Troubleshooting  (AMIT)  research 
project  design  team  to  determine  the 
appropriate  collaboration  tools  for  the 
AMIT  job  aide.  In  general,  this  strategy 
utilized  a  blend  of  research  and  user 
feedback,  balanced  by  end  use 
environment  constraints  and 
requirements,  to  create  a  usable  and 
useful  collaboration  capability  for  the 
Air  Force  maintenance  technician. 

Introduction 

In  most  jobs,  the  ability  to  collaborate 
with  other  people  at  specific  points 
during  a  task  is  essential  in  completing 
the  task  correctly.  Even  cognitively 
simple  tasks  that  are  typically 
accomplished  by  a  single  individual  may 


require  communication  at  the  beginning 
to  fully  understand  the  requirements  and 
goals  essential  to  successful  task 
completion.  The  requirement  for 
collaboration  increases  with  the 
complexity  of  the  task.  In  the  case  of 
aircraft  maintenance  troubleshooting, 
where  large  amounts  of  knowledge  and 
skill  are  required,  few  technicians 
possess  all  the  knowledge  necessary  to 
solve  all  possible  scenarios.  In  fact,  as 
part  of  our  research,  expert  maintainers 
have  stated  that  contacting  another 
technician  for  advice  is  a  logical  (and 
often  necessary)  step  in  solving  very 
difficult  problems. 

The  Air  Force  Research  Laboratory  has 
embarked  on  a  research  project 
exploring  ways  to  develop  an  advanced 
job  aid  for  flightline  maintenance  to  help 
technicians  solve  difficult  problems. 
The  Aircraft  Maintenance  Intuitive 
Troubleshooting  (AMIT)  project  is  a 
three  -year  Proof-of-Concept  effort 
headed  by  the  Logistics  Readiness 
Branch,  Warfighter  Readiness  Research 
Division,  Human  Effectiveness 
Directorate  (HEAL)  through  NCI 
Information  Systems,  Inc,  with  the 
support  of  its  subcontractor,  the 
University  of  Dayton  Research  Institute 
(UDRI).  As  a  job  aid  for  the  flightline 
maintenance  technician,  AMIT’s  goal  is 
to  support  aircraft  maintenance 
technicians  by  providing  them  the  right 
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information,  appropriate  to  their 
experience  level,  so  they  will  more 
efficiently  and  effectively  complete  their 
task.  A  successful  AMIT  project  will  be 
achieved  when  a  novice  maintained  s 
performance  can  be  brought  to  (or  near) 
the  level  of  an  expert  maintainer,  and 
when  an  expert  maintainer  has 
demonstrated  an  improvement  in 
accuracy  and  time  to  complete  when 
using  the  AMIT  capability.  These 
performance  improvements  at  the 
individual  level  will  aggregate  to  an 
improved  level  of  maintenance 
performance  at  the  squadron,  wing,  and 
Air  Force  levels.  AMIT  will  accomplish 
its  goal  through  a  number  of  methods, 
such  as  rendering  of  technical  data  and 
instructions,  autonomic  search  queries, 
and  capturing  of  maintenance  actions 
and  strategies  for  reuse.  However, 
another  important  aspect  we  discovered 
through  our  research  was  the  capability 
to  collaborate  with  other  technicians, 
especially  when  a  difficult  repair  is  not 
covered  by  the  maintenance  technical 
data. 

In  flightline  maintenance,  collaboration 
is  an  important  factor  in  the  maintainer’ s 
quest  to  complete  a  difficult  or  unusual 
repair.  Technology  provides  a  multitude 
of  potential  collaborative  tools  and 
techniques,  and  this  must  be  balanced 
against  the  requirement  to  leverage 
and/or  support  maintainers’  existing 
interaction  skills,  rather  than  requiring 
them  to  adapt  to  the  technology.  This 
paper  discusses  the  definition  of 
collaboration,  a  high-level  classification 
of  collaboration  types,  followed  by  a 
discussion  of  the  strategy  used  by  the 
AMIT  design  team  to  determine  the 
appropriate  collaboration  tools  for  the 
AMIT  job  aide.  In  general,  this  strategy 
utilized  a  blend  of  research  and  user 


feedback,  balanced  by  end  use 
environment  constraints  and  require¬ 
ments,  to  create  a  usable  and  useful 
system  for  the  Air  Force  maintenance 
technician. 

Collaboration  Definition 

Collaboration  involves  the  exchange  of 
information  between  two  or  more 
parties.  A  foundation  for  collaborative 
system  design  is  cited  in  Tolmie  and 
Boyle  (2000),  who  identify  the  two 
components  necessary  for  a  collabora¬ 
tive  system:  information  in  and 

information  out.  Coleman  and  Ward 
(2001)  go  further  by  suggesting  that  the 
benefits  of  collaboration  will  be  realized 
only  if  three  elements  converge: 
context,  content,  and  process.  Without  a 
common  context,  that  is,  a  common 
conceptual  framework  and  language,  it  is 
difficult  to  transfer  information  ( content ) 
through  interpersonal  interactions. 
There  must  also  be  a  way  (process )  for 
all  participants  to  collaborate  and 
communicate  about  this  specific,  shared 
content.  In  short,  any  given 
collaborative  technology  must  facilitate 
the  convergence  of  content,  context  and 
process  within  the  digital  collaborative 
workspace. 

Once  the  variables  have  been  addressed, 
the  goal  of  collaboration  is  to  facilitate 
understanding  and  the  effective  transfer 
of  knowledge.  Espinosa  et  al  (2002)  use 
the  term  shared  mental  models  to 
describe  the  combined  benefits  of  shared 
task  knowledge  and  shared  knowledge 
of  the  participants.  Dourish  and  Berlotti 
(1991)  help  provide  a  framework  for 
this,  specifically,  whether  the 
information  is  explicitly  generated, 
directed  and  separate  from  the  shared 
work  object;  or  passively  collected  and 
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distributed,  and  presented  in  the  same 
shared  work  space  as  the  object  of 
collaboration.  The  latter  mechanisms 
are  frequently  restricted  to  systems  in 
which  all  collaborators  are  virtually  co¬ 
present  and  working  at  the  same  time. 
However,  Dourish  and  Berlotti  feel  there 
is  no  need  for  this  restriction,  but  rather 
emphasize  the  object  of  collaboration  as 
the  driving  factor. 

In  AMIT’s  case,  the  participants  are  pre¬ 
defined  (the  maintainer  is  collaborating 
with  an  expert),  so  the  focus  can  be 
directed  to  task  knowledge.  Thus,  as  per 
Espinosa’s  criteria,  collaboration  via 
AMIT  can  focus  on  the  characteristics  of 
the  task  that  both  parties  need  to  know 
about.  Further,  the  criteria  presented  by 
Dourish  and  Berlotti  imply  that  any 
collaborative  capability  used  by  AMIT 
needs  to  incorporate  information  about 
the  aircraft  in  some  form  for  use  as  a 
reference  point.  Such  shared  feedback 
overcomes  problems  with  informational 
and  role-restrictive  approaches.  In 
particular: 

1 .  Shared  feedback  reduces  the  costs  to 
individuals  of  information  product¬ 
ion  by  collecting  information  pass¬ 
ively  and  avoiding  restrictions  on 
activities. 

2.  Shared  feedback  allows  participants 
to  look  for  and  extract  the  awareness 
information  which  is  most  relevant 
to  them. 

3.  Shared  feedback  presents  awareness 
information  through  the  shared 
workspace  and  links  to  it,  so  that 
users  can 

a.  find  relevant  information  along 
with  the  shared  object,  and 

b.  browse  awareness  information 
and  the  work  object  concurrently. 


While  the  above  list  seems  to  indicate 
the  need  for  high-fidelity  during  real¬ 
time  collaboration,  appropriate  feedback 
may  not  involve  interpersonal 
communication  (e.g.,  facial  expressions, 
hand  gestures),  but  rather  information 
about  the  task.  Kraut  (2002)  states: 

The  interactions  between  the  fidelity 
of  shared  visual  space  and  the  task 
manipulations  demonstrate  the 
importance  of  understanding  the  task 
when  determining  the  value  of  a 
shared  visual  space....  [TJhe  utility 
of  a  shared  visual  space  depends  in 
part  on  the  visual  complexity  of  the 
task.  In  settings  with  many  objects  in 
a  variety  of  spatial  relationships  to 
one  another  (e.g.,  medical  setting, 
aircraft  repair),  visual  space  may  be 
particularly  important.  For  less 
complex  visual  tasks,  especially 
those  in  which  objects  and  spatial 
relationships  are  static  and  easily 
lexicalized,  an  audio-only  connection 
may  suffice. 

Categorizing  Collaborative 
Techniques 

Kraut’s  words  touch  on  the  issue  of 
practicality;  that  is,  the  need  to  quantify 
technological  capabilities  that  would 
actually  work  in  the  targeted  end-use 
environment.  To  translate  that 
quantification  into  a  system  design 
requires  codifying  the  collaboration 
events  into  differing  event  types,  each 
with  their  own  unique  needs  and 
constraints.  Three  general  categories  of 
collaboration  techniques  were  identified 
for  the  AMIT  project;  however,  they  are 
general  enough  to  provide  a  categorical 
checklist  for  any  evaluation  of  a 
collaborative  capability.  These 
categories,  described  in  detail  below, 


3 


are:  1)  Live  Transmission;  2)  Message 
Based;  and  3)  Archived  Messages. 

Type  1:  Live  Transmission:  Also 
known  as  synchronous  collaboration 
(Dourish  &  Berlotti,  1991),  live 
transmission  is  a  collaboration  event 
occurring  in  real  time  and  addressing  a 
specific  task.  The  conveyance  of 
knowledge  is  qualified  by  environmental 
and  emotional  cues,  which  may  be  of 
equal  importance  to  the  information 
conveyed.  The  communication  is  with  a 
limited  number  of  individuals,  and  time 
is  a  critical  element.  A  telephone  is  a 
typical  example  of  this  type,  of 
collaboration  event.  The  tone  of  voice 
and  background  sounds  are  cues 
additional  to  any  technical  information 
conveyed. 

The  following  requirements  were 
identified  for  a  Live  Transmission 
collaborative  capability: 

•  Presence  awareness  of  who  can  be 
currently  contacted. 

•  Awareness  of  what  mediums  can  be 
used  with  the  contact. 

•  Routing  engines  that  deliver  the 
transmission  to  the  correct  party. 

•  Limited  need:  not  likely  required  for 
a  high  percentage  of  collaborative 
situations. 

•  Audio  /Video  (A/V)  mediums  are 
preferred,  text  based  systems  such 
Instant  Messaging  are  less  desirable. 

•  Broader  bandwidth  to  support  real¬ 
time  speeds  and  the  wider  bandwidth 
needs  of  A/V  mediums. 

•  Quality  of  A/V  medium  depends  on 
the  cues  needing  to  be  conveyed. 
Audio  will  suffice  when  video  cues 
are  not  required. 

•  Contact  pool  can  be  smaller  for  type 
1  because  the  information  needed  in 


real  time  is  more  specific  than  near- 
real  time. 

•  Do  not  require  storage  systems. 

Type  2:  Message  Based:  A  message- 
based  collaboration  event  is  an  exchange 
of  messages  in  near  real  time  for  the 
purpose  of  sharing  knowledge  about  a 
particular  task.  The  knowledge  can  be  in 
a  question-answer  format  or  purely 
informational.  The  assumption  in  a 
message-based  medium  is  that  a 
response  will  come,  though  not  always 
immediately,  and  a  limited  number  of 
responding  individuals  will  be 
participating.  When  time  is  not  a  critical 
element  and  the  collaborating  parties 
have  multiple  tasks  to  accomplish, 
message-based  collaboration  may  be  the 
most  practical  form  of  collaboration. 
Further,  although  not  in  real  time,  this 
form  of  collaboration  can  include  audio 
and  visual  cues.  For  example,  emailing 
a  recorded  audio  of  clicking  noise  or  a 
picture  of  corrosion  illustrates  a  common 
message  based  collaboration  technique. 
An  e-mail  is  sent  to  a  limited  number  of 
individuals  and  a  timely  (but  not 
immediate)  response  is  generally 
expected.  When  an  e-mail  is  broadcast 
to  a  group,  and  is  purely  informational 
without  an  expected  response,  it  is 
classified  as  an  archived  message  event 
(see  below). 

Message  based  systems  need  the 
following: 

•  Presence  awareness  of  who  can  be 
currently  contacted. 

•  Awareness  of  what  mediums  can  be 
used  with  the  contact. 

•  Routing  engines  that  deliver 
messages  to  the  correct  party. 

•  High  need  and  practicality:  this 
collaborative  technique  should  be 
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accessible  from  any  process  and 
context. 

•  Do  not  require  real-time  com¬ 
munication,  but  the  closer  to  real 
time,  the  more  efficient.  The 
slower/older  the  communication,  the 
less  efficient. 

•  Is  task  oriented. 

•  Ideally  the  message  should  be 
supported  by  A/V  mediums,  but  text 
will  suffice  for  most  message  needs. 

•  May  require  temporary  storage 
systems. 

Type  3:  Archived  Messages:  An 
archived  message  is  an  authored 
collaboration  event  intended  for  a  wide 
audience.  An  archived  copy  of  a  Live  or 
Message  Based  event  that  has  use 
beyond  a  single  task  can  be  considered 
an  Archived,  or  Type  3  event.  An 
archive’s  purpose  is  to  be  a  searchable 
knowledge  repository  that  is  not  specific 
to  an  individual  task.  Information  that  is 
not  generalizable,  or  reusable  for  other 
purposes,  do  not  need  to  be  archived. 
Another  goal  is  to  communicate 
knowledge  across  time,  such  as  across 
shifts.  A  logbook  is  an  example  of  a 
Type  3  collaboration  event.  A  logbook 
provides  information  to  all  of  its  readers 
the  accomplishments  of  a  task  that  may 
have  relevance  to  other  tasks.  In  most 
cases,  text  will  suffice  for  storage  and 
retrieval.  Text  extracted  from  a 
conversation  is  sufficient  if  the  cues  in 
the  conversation  are  not  important.  In 
cases  where  cues  are  important,  only  the 
cue  needs  to  be  stored  and  indexed.  A 
two-hour  conversation  may  only  contain 
one  still  picture  and  a  small  amount  of 
text  for  reuse  of  the  knowledge 
conveyed. 

Archived  Message  systems  need  the 
following: 


•  Storage  systems  with  maintenance 
support. 

•  Access  by  wide  range  of  individuals. 

•  Identity  systems  to  control  access  to 
the  information,  and  to  identify 
sources  storing  data  for  relevance 
sorting. 

•  Indexing  systems  to  retrieve  relevant 
information. 

•  Search  systems  to  query  the  index. 

•  Relevance  engine  to  qualify  the 
information  retrieved  from  a  search. 

•  Limited  availability.  The  information 
is  by  request,  and  while  high 
availability  is  preferred,  it  is  not 
required. 

•  Require  only  enough  bandwidth  to 
transmit  data  one-way,  and  in  near- 
real  time. 

•  Copies  of  the  data  store  are  sufficient 
for  most  tasks. 

Collaboration  as  Applied  to  AMIT 

As  stated  above,  the  AMIT  job  aid  is 
intended  to  improve  the  maintainer’s 
ability  to  successfully  accomplish  the 
task  at  hand:  fixing  the  aircraft.  Part  of 
this  improvement  involves  providing 
capabilities  that  were  either  a)  not 
previously  available,  or  b)  available  in  a 
format  that  was  not  optimal.  The 
maintainer’s  need  and  ability  to 
collaborate  overlaps  both  a  and  b.  It  is 
the  goal  of  AMIT  to  provide  a 
collaborative  capability  that  is  always 
available  at  the  point-of-maintenance,  as 
well  as  optimized  for  the  maintenance 
environment. 

In  general,  a  maintainer  will  collaborate 
when  the  tech  data  does  not  provide 
sufficient  information  for  a  successful 
repair,  and  the  strategy  adopted  by  the 
maintainer  to  collaborate  depends  on 
what  is  available.  Current  modes  of 
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collaboration  involve  common  com¬ 
munication  technologies,  for  example,  a 
phone  call  or  email.  However,  the  state- 
of-the-art  in  collaborative  technologies 
clearly  includes  additional,  “higher  tech” 
capabilities,  such  as  instant  messaging, 
real-time  digital  video,  and  shared 
workspaces.  A  primary  challenge  faced 
by  the  AMIT  team  was  to  evaluate  and 
select  the  collaborative  technology(s) 
most  appropriate  to  the  flightline 
maintenance  environment. 

The  goal  of  “allowing  a  subject  matter 
expert  to  look  over  the  technician’s 
shoulder”  seems  reasonable;  however, 
technology  provides  many  paths  to 
accomplish  this.  As  the  AMIT  design 
team  identified  several  candidate 
technologies  with  disparate  capabilities, 
the  need  for  an  Aircraft  Maintenance 
User  Group  (AMUG)  for  AMIT  was 
identified  to  evaluate  the  practicality  of 
these  technologies.  The  AMUG 
convened  for  a  three  day  period  to 
specifically  address  AMIT  design  issues, 
including  collaboration.  Within  the 
topic  of  collaboration,  the  AMIT  design 
team  sought  to  identify  the  following 
information: 

•  The  parties  involved  during  coll¬ 
aboration, 

•  Information  that  is  exchanged 
(format  and  type), 

•  Current  collaborative  techniques  and 
practices, 

•  Archive  and  retrieval  requirements, 
and 

•  Environmental  issues. 

The  parties  involved  with  collaboration 
were  identified  as  the  maintainer  and  a 
subject  matter  expert  (SME).  The  SME 
may  be  a  technical  representative  located 
on-site,  or  an  experienced  maintainer 


located  at  a  different  base.  The  format 
of  information  exchanged  can  be 
assumed  to  have  a  visual  component,  or 
at  the  very  least,  a  means  to  reference 
the  technical  manuals.  However, 
technology  allows  the  further  addition  of 
informational  components  to  the  point  of 
excess,  or  irrelevant,  information.  Thus, 
the  type  of  information  required  to 
successfully  collaborate  must  also  be 
addressed;  for  example,  while 
technology  can  allow  a  video  image  the 
of  the  other  party’s  face  during  a 
collaborative  session,  the  need  for  this 
does  not  exist.  Discussions  of  location 
issues  assumed  that  the  technician’s 
portion  of  a  collaborative  session  would 
be  conducted  via  the  AMIT  tool,  and 
thus  be  carried  out  at  the  aircraft. 
Currently,  in  order  to  accomplish  any 
form  of  collaboration,  the  technician 
must  leave  the  aircraft  and  access  the 
collaborative  medium  at  a  different 
location,  consuming  time  for  transit  and 
communication  away  from  the  flightline 
—  time  that  could  ideally  be  spent  at  the 
aircraft.  Because  collaborative  sessions 
are  not  officially  documented  in  any 
form,  the  issue  of  archiving  was 
addressed.  Currently,  the  maintainer 
may  take  notes,  or  in  the  case  of  email, 
save  the  emails.  However,  no  standard 
mechanism  exists  which  would  allow 
future  maintainers  to  access  and  leverage 
useful  information  from  previous 
collaborative  sessions.  Because  the 
maintainer  will  be  collaborating  from  the 
flightline,  environmental  concerns  such 
as  ambient  noise  were  discussed. 
Finally,  the  issue  of  Live  Transmission 
vs.  Message  Based  vs.  Archived 
Messages  collaboration  was  discussed, 
and  the  practicality  of  each  was 
evaluated. 
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Discussing  collaboration  requirements 
with  end-users  provided  real-world 
context  to  an  array  of  candidate 
technologies  and  thus  greatly  helped  to 
scope  the  selection  of  these  technologies 
for  use  in  the  AMIT  project.  To  begin, 
the  AMUG  validated  that  A/V  aids  in 
conveying  troubleshooting  data.  For 
example,  audio  was  identified  as  good 
for  avionics  repair  tasks,  while  video 
was  seen  as  more  relevant  for  radar.  The 
use  of  a  whiteboard  capability  was  seen 
as  useful  for  shift  change,  but  not  during 
an  actual  repair.  For  documentation 
purposes,  text  was  considered  acceptable 
for  archived  collaborative  sessions, 
rather  than  archiving  in  the  same  format 
that  the  actual  collaboration  session  used 
(i.e.  archiving  video  as  a  video  file). 
Users  also  agreed  that  AMIT  should 
integrate  collaborative  sessions  into  the 
logbook.  In  general,  users  indicated  a 
willingness  to  adapt  to  various 
collaborative  technologies,  depending  on 
what  is  fielded. 

Once  the  AMUG  helped  the  AMIT 
design  team  to  scope  collaboration 
needs,  a  second  round  of  evaluation  was 
conducted,  involving  rapid  prototypes  of 
various  collaborative  capabilities.  These 
Design  Consideration  Tests  (DCTs) 
were  intended  to  measure  time  and 
errors  across  the  combined  effects  of 
three  levels  of  task  complexity  and  three 
levels  of  collaborative  fidelity.  The 
three  levels  of  collaborative  fidelity 
were:  1)  Audio,  2)  Text,  and  3)  Text  + 
Visual.  The  three  levels  of  task 
complexity  were  designed  to  be 
representative  of  aircraft  maintenance, 
and  required  various  types  of 
information  for  successful  completion. 
They  were:  1)  Simple  (involving  a 
yes/no  response  to  a  question;  2) 
Moderate  (involving  acquiring  textual 


information  and  filling  out  a  form),  and 
3)  Difficult  (requiring  the  identification 
of  specific  locations  on  an  aircraft). 
Results  of  the  DCTs  verified  that,  in 
terms  of  accuracy,  increased  task 
complexity  drives  the  need  for  increased 
collaborative  fidelity.  Specifically, 
audio  was  identified  as  the  most  error- 
prone  medium  across  all  tasks. 
However,  for  task  completion  time, 
results  indicated  that  audio  was  faster 
than  the  text  and  text  +  visual 
conditions.  It  is  likely  that  audio  was 
fastest  because  it  simply  did  not  provide 
the  means  to  convey  all  necessary 
information.  The  trade-off  was  quicker, 
but  more  error  prone,  communications. 
Post-test  subjective  feedback  indicates 
that  participants  may  have  recognized 
this  trade-off,  as  they  felt  that  they 
communicated  more  clearly  with  the 
high-fidelity  collaborative  capabilities 
(Text  and  Text/Visual)  than  with  audio. 

Conclusion 

The  AMUG  sessions  and  the  DCT 
results  helped  identify  the  scope  and 
performance  issues  associated  with 
collaboration  in  the  context  of  flightline 
maintenance.  As  a  result,  the  design 
team  was  able  to  make  a  series  of  critical 
decisions  involving  the  scope  of  the 
AMIT  collaborative  capability.  First, 
the  need  for  a  visual  component  is  seen 
as  necessary,  as  is  the  need  for  a  text- 
based  messaging  capability.  Also,  the 
ability  to  transmit  voice  will  be  given  a 
high  priority,  specifically  to  be  used  in 
conjunction  with  a  visually  oriented 
collaborative  task.  The  resulting  design 
should  provide  a  collaborative  capability 
that  will  be  effective  across  maintenance 
situations,  user  experience  level,  and 
environment. 
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However,  the  road  to  making  these 
decisions  was  not  easily  traveled.  One 
of  the  biggest  challenges  faced  by  the 
AMIT  design  team  was  the  fact  that  so 
many  collaborative  technologies 
currently  exist.  The  design  team  faced 
many  confrontations  between  practical¬ 
ity  and  technology  seduction.  In  the 
long  run,  the  use  of  an  end-user  group 
(the  AMUG)  and  a  performance-based 
evaluation  of  candidate  capabilities, 
helped  greatly  in  answering  the 
questions  of  1)  what  is  truly  needed,  2) 
what  will  improve  performance,  and  3) 
what  would  the  end-users  like.  The 
answers  to  these  questions  do  not  always 
jibe,  however,  as  illustrated  by  the  mixed 
results  involving  the  DCTs.  In  the  end, 
the  design  team  weighed  the  answers  of 
each  and  made  educated  design 
decisions  that  will  provide  the  most 
benefit  in  terms  of  improved  perform¬ 
ance  and  user  acceptance. 
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